StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles

Different people speak with diverse personalized speaking styles. Although existing one-shot talking head methods have made significant progress in lip sync, natural facial expressions, and stable motions, they still cannot generate styles the final videos. To tackle this problem, we propose a style-controllable face generation framework. In nutshell, aim to attain style from an arbitrary reference video then drive portrait another piece of audio. Specifically, first develop encoder extract dynamic motion patterns encode them into code. Afterward, introduce decoder synthesize stylized animations speech content order integrate generated videos, design style-aware adaptive transformer, which enables encoded code adjust weights feed-forward layers accordingly. Thanks adaptation mechanism, can be better embedded synthesized videos during decoding. Extensive experiments demonstrate that our method is capable generating only one image audio clip while achieving authentic visual effects. Project Page: https://github.com/FuxiVirtualHuman/styletalk.